The records queue re-delivers immediately (no `retry_delay`), so classifying an
empty DNS answer / resolver outage as transient only helped if the retry waited
for propagation — otherwise all `max_retries` (5) burn in seconds and the record
DLQs as UNEXPECTED_ERROR before DNS settles. `PdsVerificationError` now carries
an optional `retryAfterSeconds`, set (to 60s) only on the transient DNS-resolution
path; the consumer passes it as `retry({ delaySeconds })`. Other transient
retries (network, 5xx, timeout) stay immediate.

Delay strategy: fixed 60s. With max_retries: 5 that spreads retries across a
~5-minute propagation window. Chosen for reconciliation with labeler #2113.

…nst SSRF (review round 1) (#2117)

* fix(aggregator): wrap unexpected XRPC errors and harden PDS egress against SSRF

Finding 17 (Sol): a non-XRPC failure resolving the labeler policy (e.g. a D1
error) escaped `handleXrpc` unwrapped, hitting workerd's bare 500 without the
CORS or `private, no-store` headers every aggregator response carries. Convert
unexpected errors to a generic 500 through the same wrapper, logging the
internal detail but never leaking it to the client. XRPCError handling is
unchanged.

SSRF follow-up: `pds-verify.ts` fetched the publisher-controlled PDS endpoint
(from a DID document) following redirects with no scheme/DNS/reserved-address
validation and no body-read deadline. Route it through the shared hardened
egress (`resolveAndValidateExternalUrl` + injected `cloudflareDohResolver`):
HTTPS-only, per-hop re-resolution under `redirect: "manual"` with a bounded
redirect count, a wall-clock deadline spanning redirects and the body read, and
fail-closed when no resolver is injected. Blocks are reported as a new permanent
`PDS_ADDRESS_BLOCKED` reason; existing 404/5xx/too-large/invalid-proof and the
transient network classification are preserved.

* fix(aggregator): retry PDS verification on resolver-infra failures, don't dead-letter

`assertFetchableUrl` collapsed every `SsrfError` from
`resolveAndValidateExternalUrl` into the permanent `PDS_ADDRESS_BLOCKED`, but
that helper also throws `SsrfError` when the injected DNS resolver itself fails
(DoH network error / SERVFAIL / timeout) — transient infrastructure, not a
disallowed publisher address. A DoH blip therefore permanently dead-lettered
legitimate records and risked mass dead-lettering during a resolver outage.

`SsrfError.code` is the same constant for both cases, so key on the message
prefix the helper uses only for the resolver-threw path (`Could not resolve
hostname:`): map that to the transient `PDS_NETWORK_ERROR` (retried), and keep
genuine address blocks — private/reserved IP, blocked host, scheme — as the
permanent `PDS_ADDRESS_BLOCKED`. A test asserts the transient mapping so a future
wording change in the shared core helper breaks loudly rather than silently
mis-classifying. Also adds a redirect-hop-limit boundary test.

* fix(aggregator): retry PDS verification on empty DNS answer, don't dead-letter

An empty resolver answer (NXDOMAIN / NOERROR-NODATA / CNAME-only) makes
`resolveAndValidateExternalUrl` throw `SsrfError("Hostname resolved to no
addresses")`, which the classifier mapped to the permanent PDS_ADDRESS_BLOCKED
— so a host mid-DNS-propagation lost the record to a dead-letter. Treat the
empty-answer message as a transient resolution failure (PDS_NETWORK_ERROR,
retried) alongside the existing resolver-threw case; a genuinely-gone host
still dead-letters via retry exhaustion. Genuine private/reserved/blocked-host
addresses stay permanent. Keyed on the shared helper's message as a module
constant with a test, matching the resolver-failure-prefix approach.

* fix(aggregator): delay re-delivery on transient DNS-resolution failure

The records queue re-delivers immediately (no `retry_delay`), so classifying an
empty DNS answer / resolver outage as transient only helped if the retry waited
for propagation — otherwise all `max_retries` (5) burn in seconds and the record
DLQs as UNEXPECTED_ERROR before DNS settles. `PdsVerificationError` now carries
an optional `retryAfterSeconds`, set (to 60s) only on the transient DNS-resolution
path; the consumer passes it as `retry({ delaySeconds })`. Other transient
retries (network, 5xx, timeout) stay immediate.

Delay strategy: fixed 60s. With max_retries: 5 that spreads retries across a
~5-minute propagation window. Chosen for reconciliation with labeler #2113.

After merging the integration branch (which added #2113's transient-DNS retry
backoff), the applyDiscoveryDelete docstring still said a failed negation
propagates to classifyDiscoveryError. Since the Blocker-2 fix, the delete
mutation phase has its own always-retry catch — update the comment to match.

No behavior change.

…w round 1) (#2115)

* fix(labeler): redrive the subscription-DO notify on console mutation replay

A transient afterCommit failure after a label commits leaves the row
publication_pending=1 and drops the live broadcast; the guard replay branch did
not re-drive the notify, so a client retry never recovered it. The label-issue
and emergency replay branches now re-fire the post-commit notify (keyed on the
committed action), giving retries a cheap path back to a live broadcast. The
reconciliation publication-pending sweep remains the durable backstop.

Refs Sol finding 7.

* fix(labeler): re-sign retired-key labels on WebSocket subscription replay

After a routine key rotation the DID document publishes only the new key, but
retained rows keep their old-key signatures; queryLabels re-signs lazily on read
while the subscribeLabels replay path sent rows unchanged, so a fresh aggregator
rejected the first old-key frame and could not advance. The replay reader now
reuses queryLabels' lazy re-sign helper (exported from query-labels), bringing
stale-key rows onto the active key before framing and persisting the result so
the work is not repeated per connection. Sequence and ordering are preserved; a
signing pause throws rather than serving an unverifiable frame.

Refs Sol finding 5.

* fix(labeler): publish automated assessment labels live, with a reconciliation backstop

Automated pending/outcome/block/negation labels committed without ever notifying
the subscription DO, so a connected aggregator received nothing until it
reconnected or an unrelated notification arrived. The orchestrator now issues
finalization labels publication_pending=1 and broadcasts each to the same
subscription-DO publisher the console path uses, off the commit. A new
publication-pending sweep in the reconciliation cron re-drives any notify that
was dropped (assessment or console path alike) — the durable guarantee that a
stranded row reaches subscribers and stops blocking the next key rotation.

Refs Sol findings 4 and 7.

* fix(labeler): guard the finalization CAS on signing state so a rotation pause can't strand a run

If signing paused between finalization prep and the batch commit, the guarded
label INSERTs no-oped while the unguarded assessment CAS still committed terminal
state — leaving the run terminal with its pending-negation, outcome, and block
labels missing, and the Workflow retry seeing terminal and never repairing them.
The CAS now shares the same signing-state predicate as the issuance statements,
so the whole db.batch is all-or-nothing: a mid-batch pause no-ops the CAS too,
the run stays running, and the retry re-runs finalization after signing resumes.

Refs Sol finding 6.

* perf(labeler): index issued_labels for the publication-pending sweep

The reconciliation sweep filtered issued_labels by publication_pending ordered
by sequence with no covering index, full-scanning a monotonically growing table
every cron tick — a D1 query-timeout there would strand pending rows and block
the very rotation drain the sweep exists to unblock. Adds migration 0011 with a
partial covering index on (sequence, cts) WHERE publication_pending = 1, and a
query-plan guard so the sweep can't silently regress to a full scan.

Follow-up to emdashbot review on #2115.

* docs(labeler): drop the stale mid-batch signing-flip gap from finalize

The F6 fix guards the finalization CAS on the same signing-state predicate as
the label issuances, so the race-analysis comment's first bullet (claiming the
CAS is unguarded and a flip could commit terminal state with labels suppressed)
is no longer true. Replaces it with the now-closed description and keeps the two
genuinely-remaining narrower gaps.

Follow-up to emdashbot review on #2115.

* fix(labeler): publish discovery-consumer labels live, with the sweep backstop

The #4 fix gave the orchestrator path live publication but missed the
discovery-consumer path: its initial assessment-pending issuance and its
deletion negations committed publication_pending=0 with no subscription-DO
broadcast, so connected subscribers never received them and the new
reconciliation sweep (pending=1 only) could not recover them. Both issuances now
take the same subscription-DO publisher the orchestrator uses — publication_pending=1
on commit plus a best-effort post-commit notify (a dropped broadcast never fails
the discovery message; the sweep re-drives it). The rotation-drain contract is
unchanged: these pending rows drain exactly like the orchestrator's.

Follow-up to Sol review on #2115.

* fix(labeler): negate a deleted run's pending label before cancelling it

The delete path cancelled pending/running runs to terminal 'cancelled' before
issuing their assessment-pending negations. If issuance was paused (signing
rotation) the negation threw and the message retried, but on redelivery
listNonTerminalAssessmentsForUri excluded the already-cancelled runs — so there
was nothing left to negate, the message acked, and the active assessment-pending
label survived a deleted release forever. Each run's negation is now issued
before its terminal cancellation, so a paused/failed negation leaves the run
non-terminal and re-discoverable on redelivery; a run is retired only after its
negation commits, and the message cannot ack while any pending/running run is
un-negated. No active assessment-pending survives an acked delete.

Follow-up to Sol review on #2115.

* fix(labeler): guard the finalization CAS on the subject not being tombstoned

Finalization's isSubjectCurrent re-check is a separate read from its commit CAS, a
TOCTOU: a delete that tombstoned the subject in between still let the CAS commit
'running -> outcome' plus positive outcome/block labels, then the delete's own
cancel CAS conflicted and was swallowed — leaving live block labels on a deleted
release the evaluator would still honor. The finalization CAS now carries a
not-deleted predicate on the run's subject (same all-or-nothing batch idiom as
the F6 signing guard), so a tombstone landing before the batch no-ops the CAS
and every label gated on the outcome state; finalization retries and stales the
run out, never labelling a deleted subject.

Follow-up to Sol round-3 review on #2115.

* fix(labeler): retry (never dead-letter+ack) a delete-path mutation failure

The 'a negation throw prevents ack' invariant only held for recognized
issuance-unavailable errors; an unexpected signing/D1 failure during the delete's
tombstone/negate/cancel fell through to dead-letter+ack (the create-path policy),
leaving the run non-terminal with its assessment-pending label live on a deleted
subject forever. The delete handler now splits the absence-verification phase
(classified like create: transient retries, forged/permanent dead-letters) from
the mutation phase, which ALWAYS retries on failure so the label can never be
left live on an acked delete. A genuinely permanent fault exhausts to the DLQ via
max_retries — acceptable versus acking a stranded label.

Follow-up to Sol round-3 review on #2115.

* docs(labeler): correct the delete-mutation retry reference post-merge

After merging the integration branch (which added #2113's transient-DNS retry
backoff), the applyDiscoveryDelete docstring still said a failed negation
propagates to classifyDiscoveryError. Since the Blocker-2 fix, the delete
mutation phase has its own always-retry catch — update the comment to match.

No behavior change.

* fix(labeler): guard the initial pending-label issuance against a concurrent delete

The create path committed a recreated run as pending and THEN issued its positive
assessment-pending with no commit-time guard. A concurrent delete could, in the
gap, tombstone the subject, negate + cancel the (still-positive-less) run, and
ack; the create then committed its positive at a higher sequence, winning label
reduction — a live pending label on a tombstoned, cancelled subject. (Queue
invocations overlap; max_concurrency is unset.)

The initial issuance now goes through buildIssuanceStatements gated atomically on
BOTH the run still being pending AND the exact (uri,cid) subject still undeleted
(the same guard idiom as Blocker 1's finalization CAS). On a guard miss the label
no-ops: the issuance is obsolete, so the create neither publishes nor dispatches a
Workflow for a label that never committed; a non-persist not explained by the
guard (a signing flip) still retries. With this and Blocker 1, every automated-
positive issuance site in the discovery consumer is now delete-guarded (the
delete-path negation is negative and exempt).

Follow-up to Sol round-4 review on #2115.

* fix(labeler): guard the operator-rerun pending label against a concurrent delete

The operator rerun issued its positive assessment-pending with no commit-time
guard and while its run was still 'observed' (advance to 'pending' is deferred),
opening two integrity races with a concurrent discovery-delete (queue + HTTP
overlap; max_concurrency unset):
  (a) ordering: the rerun commits its run+pending after a delete tombstoned and
      snapshotted the non-terminal runs, so the delete never negates it and the
      positive survives on the tombstoned subject;
  (b) observed gap: the delete-path negation only fired for pending/running runs,
      so a delete that saw the rerun's 'observed' run cancelled it WITHOUT negating
      its already-live positive.

Fix (a): gate the rerun's assessment-pending issuance on requireSubjectNotDeleted
+ requireAssessmentState (threaded through prepareAutomatedLabelIssuance); on a
miss the label no-ops, assertIssuancePersisted aborts before the deferred tail, so
nothing is published, dispatched, or advanced. Fix (b): the delete-path negation
now fires for any non-terminal run that committed a positive pending (keyed on the
committed positive, not lifecycle state) — no dangling negations, no interleaving
where a cancelled/deleted run keeps a live positive. With Blocker 1 and the round-4
discovery guard, every positive assessment-pending issuance across the discovery
and console paths is now delete-safe.

Follow-up to adversary review on #2115.

* fix(labeler): structural delete-generation guard closing the create/verify/rerun-vs-delete race class

Adds a monotonic delete_generation to subjects (migration 0013). Every delete
increments it; create/verify/rerun capture it before reading state/verifying and
CAS-guard their subject-undelete, run creation, and label issuance on the
generation not having advanced. A decision made before a delete is rejected
obsolete; one that captured the post-delete generation still works (republish).

Closes three race seams: (1) createSubject un-delete is generation-gated so a
stale verify can't resurrect a deleted subject; (2) the delete cleanup now scans
pending-bearing runs INCLUDING terminal stale and negates any committed positive;
(3) the console rerun's run creation is generation-gated so no orphan observed run.

Implementation checkpoint; barrier tests for each seam + a republish test follow.

Refs Sol round-5 (systemic close).

* test(labeler): barrier tests for the delete-generation close (three seams + republish)

Seam 1: a stale verify (barrier during verify, full delete completes) cannot
resurrect the tombstoned subject — createSubject's generation-guarded undelete
no-ops, no run, no positive, no dispatch. Seam 2: the delete negates a terminal
stale run's stranded positive (widened scan). Seam 3: a rerun after a concurrent
tombstone leaves no orphan observed operator run. Plus a delete-then-republish
test proving a new revision after a delete assesses cleanly (the generation does
not over-block). Each verified failing pre-fix.

Refs Sol round-5 (systemic close).